Principles of Riemannian Geometry in Neural Networks

This study deals with neural networks in the sense of geometric transformations acting on the coordinate representation of the underlying data manifold which the data is sampled from. It forms part of an attempt to construct a formalized general theory of neural networks in the setting of Riemannian geometry. From this perspective, the following theoretical results are developed and proven for feedforward networks. First it is shown that residual neural networks are nite di erence approximations to dynamical systems of rst order di erential equations, as opposed to ordinary networks that are static. This implies that the network is learning systems of di erential equations governing the coordinate transformations that represent the data. Second it is shown that a closed form solution of the metric tensor on the underlying data manifold can be found by backpropagating the coordinate representations learned by the neural network itself. This is formulated in a formal abstract sense as a sequence of Lie group actions on the metric bre space in the principal and associated bundles on the data manifold. Toy experiments were run to con rm parts of the proposed theory, as well as to provide intuitions as to how neural networks operate on data.